Systems and methods for developing computer algorithm solutions by conducting competitions

ABSTRACT

This invention relates to a method and apparatus for providing a coding competition. In one embodiment, a method for providing a coding competition includes transmitting a coding problem to contestants, receiving computer code created by a contestant, determining a response of the computer code to test data, and evaluating the response of the computer code to the test data. In another embodiment, a method for evaluating a skill level of a contestant includes electronically communicating a coding problem to contestants, electronically receiving a software program in response to the coding problem from one of the contestants, evaluating the received software program, awarding points to the contestant based on the received software program, and determining a rating for the contestant for the competition based on the number of points awarded to the contestant.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of and claims priority toco-pending U.S. patent application Ser. No. 11/800,432, entitled“Systems and Methods for Coding Competitions” and filed on May 4, 2007,which is a continuation of and claims priority to U.S. patentapplication Ser. No. 11/429,648 (now U.S. Pat. No. 7,311,595) and Ser.No. 11/193,019 (now U.S. Pat. No. 7,300,346), both entitled “Systems andMethods for Coding Competitions” and filed on May 5, 2006 and Jul. 29,2005 respectively. U.S. patent application Ser. No. 11/193,019 is acontinuation of and claims priority to then U.S. patent application Ser.No. 10/378,539, now U.S. Pat. No. 6,984,177, entitled “Method and Systemfor Communicating Programmer Information to Potential Employers” filedon Feb. 28, 2003, which is a divisional application of and claimspriority to then U.S. patent application Ser. No. 10/041,393, now U.S.Pat. No. 6,569,012, entitled “Systems and Methods for CodingCompetitions” filed on Jan. 8, 2002, which claims priority to U.S.provisional patent application Ser. No. 60/260,610, filed Jan. 9, 2001.This application is also related to U.S. patent application Ser. No.10/377,343, now U.S. Pat. No. 6,824,462, entitled “Method and System forEvaluating Skills of Contestants in Online Coding Competitions” filed onFeb. 28, 2003, U.S. patent application Ser. No. 10/377,344, now U.S.Pat. No. 6,761,631, entitled “Apparatus and System for FacilitatingOnline Coding Competitions” filed on Feb. 28, 2003, U.S. patentapplication Ser. No. 12/338,149, now U.S. Pat. No. 8,137,172, entitled“System and Method for Programming Tournaments” filed on Dec. 18, 2008,and U.S. patent application Ser. No. 12/338,132, now U.S. Pat. No.8,021,221, entitled “System and Method for Conducting ProgrammingCompetitions Using Aliases” filed Dec. 18, 2008.

TECHNICAL FIELD

This invention relates to computer-based methods and systems forcompeting and, more particularly, to methods and systems providingcoding competitions.

BACKGROUND INFORMATION

In the United States and elsewhere, computers have become part ofpeople's everyday lives, both in the workplace and in personalendeavors. This is because a general purpose computer can be programmedto run a variety of software programs each providing differentprocessing and networking functions. Computer programmers developcomputer code. Some companies hire large numbers of computer programmersto develop code on the company's behalf.

Like many other activities, coding requires a collection of skills. Forexample, some coders are extremely skilled, and, for example, areremarkably accurate and efficient. Professional certifications provideinformation about the base skills possessed by an individual, butgenerally there is no objective way to easily distinguish highly-skilledcoders from average coders. This lack of objective measure means thathighly-skilled coders may not be compensated for their true worth. Inaddition, in the United States and elsewhere, workers considered to be“professionals,” such as lawyers, accountants, doctors, and managers,command higher salaries than the general population. Meanwhile, highlyskilled programmers, who typically have good education, significantresponsibility, and relatively unique abilities, are treated likeproduction and administrative staff, rather than professionals. This isalso true for computer coders in other disciplines (e.g. hardwaredesign, and other engineering disciplines).

In the hiring process, firms typically subject coders to an interviewprocess that may even include some actual testing of candidates' codingskills. But it is very difficult for a company, even with some skillstesting, to definitively identify those programmers who have thegreatest skills, are the most accurate and efficient, and so on. Thismakes it difficult for a company that wants to hire premier coders toidentify and hire them, and it makes it more difficult for premiercoders to be hired at the compensation levels that they may deserve.

SUMMARY OF THE INVENTION

A need therefore exists for mechanisms for skilled coders to buildmarketability and raise their income to reflect the value their workproduct creates. In general, in one aspect of the invention, systems andmethods for providing coding competitions address these and other goals.Through the provision of computer-based coding competitions, successfulcoding competition participants can be distinguished from other coders,which benefits both the coders and their employers. Recognizably bettercoders can command greater salaries, and employers have an objectivemeans to compare candidates and find good coders. In some embodiments,good coders have incentives to participate as competitions provide oneor more of entertainment, awards, and future employment opportunities.

In some embodiments, entities that wish to gain access to skilled coderssponsor the coding competitions. In some embodiments, a rating system isused to provide a metric that can be used to evaluate a programmer'sproven skills, knowledge, and ability.

In various embodiments, coding competitions are provided “on-line,” inthe sense that they are provided using computers that communicate over anetwork. Coding competition participants, also referred to ascontestants, use a client computer that communicates with a competitionserver. The contestants may be located anywhere in the world if (forexample) the network is a global network such as the Internet, or thecoding competition participants may be located nearby, even in the sameroom, if (for example) the network is a local area network.

In some embodiments, as part of a coding competition, a server computerprovides to one or more contestants a computer coding problem (e.g. atest question) that requires each contestant to develop computer code.The computer code may be, for example, an entire computer applicationprogram or a large or small portion of a computer application program(e.g. a subroutine or a subsystem, one or more objects, etc.). The codemay be written in a specific programming language (e.g. C++, Java,C^(#), Fortran, perl, etc.), or may be coded in one of a choice ofprogramming languages or coding formats.

The coding problem may require design and or analysis of the problem inaddition to coding. The computer code may also be for something otherthan a computer application program; for example, it may be a design fora computer program, or it may be an electronic circuit design orimplementation, or for another engineering design and/or implementation.The computer code may be a model that will be forward engineered into amachine readable format for testing purposes. The code developed by acontestant is generally referred to as computer code, regardless ofwhether it is all or a portion of an application program, or somethingelse.

In some embodiments, portions of the coding competitions take place invirtual “rooms” in which two or more contestants compete. Eachcontestant in a virtual room receives a coding problem or problems,which problems may be of varying difficulty. Contestants write computercode in response to one or more of the presented coding problem(s). Whena contestant's code is ready, the respective client computer transmitsthe contestant's computer code to a server for evaluation. The time thecontestant took to complete the computer code may be recorded.

In some embodiments, the server evaluates the computer code, and assignsa number of points to the contestants' submission. In some embodiments,the competition includes multiple rounds in which the server evaluatescontestant computer code for each round. The server may also determine askill level rating or ratings for the contestant. The determination ofskill level of contestants may, for example, result in the awarding of aprize to a contestant, present employment opportunities to thecontestant, attract sponsorship for the competition, and present hiringopportunities to the sponsors of the competition.

In some embodiments, contestants submit the requested computer coderesponsive to the computer coding problems in a first phase of thecompetition, and the response time of the contestants is recorded. Thecomputer code written by the contestants during the first phase of thecompetition is verified during a subsequent phase of the competition.The submitted code can be verified using predetermined test data asinput. In addition, each contestant may have an opportunity to reviewthe computer code submitted by other contestants, and to submit testdata to be provided as input to the other contestants' computer code forverification of that code. In this way, contestants can challenge thework of the other contestants.

In one embodiment, one or more competitions are held among programmersin a forum setting, analogous to a spectator sport. The contestants arelocated near each other, and spectators can observe the competition, andwatch the programmers in action. There are displays and other aids toenhance the spectator enjoyment of the competition. In anotherembodiment, in which the competition takes place via a global computernetwork such as the Internet, the competitions take place in a “virtualforum” setting, in which spectators can observe, via the computernetwork, the activity in the virtual room or rooms. In some embodiments,spectators (either locally or via the network) can not only observe acoding competition, but can participate, by submitting test data to beused as input to the competitors' submissions.

In one aspect of the invention, a method for providing a codingcompetition includes the steps of transmitting a coding problem tocontestants over a communications network and receiving computer codecreated by one or more of the contestants in response to the transmittedcoding problem over the communications network. The method also includesthe steps of determining a response of the received computer code totest data submitted by a party other than the contestant and evaluatingthe determined response, where that party may be, for example, a systemadministrator or server, a judge, another contestant, a competitionspectator, and so forth.

In one embodiment, the method includes the step of providing the testdata as input to reference code and comparing output of the referencecode to output of the received computer code. In one embodiment, thereceived computer code may be evaluated based on such factors as thecorrectness of the determined the response, the accuracy of thedetermined response, and the speed of the determined response. Pointsmay be awarded to a contestant based on the evaluation of the determinedresponse of the computer code to the test data. A rating may also beassigned to the contestant based on the points awarded, and also basedon prior competition performance. In one embodiment, a contestant isassigned to a division, based on the contestant's rating. For example,the contestant may be assigned to one of a first division forcontestants who have previously attained a first rating or greater, anda second division for contestants who have a rating substantially equalto zero or a rating below a predetermined division rating. There may bea third, fourth, and divisions, etc., depending on the number ofcontestants and the spread in skill level. The use of divisions keepsthe competition interesting to the contestants.

In another aspect, a method for evaluating a skill level of a contestantincludes the steps of electronically communicating a coding problem tocontestants during a competition and electronically receiving computercode in response to the coding problem from one of the contestantsduring the competition. The method also includes the steps ofautomatically evaluating the received computer code and awarding pointsto the contestant based on the received computer code. The methodadditionally includes determining a rating for the contestant for thecompetition based on the number of points awarded to the contestant.

In some embodiments, the method includes determining an average ratingfor the competition based on previous ratings of the competition and atotal number of contestants receiving the coding problem. The method canalso include determining a competition factor for the competition basedon previous ratings of the contestants, a total number of contestantsreceiving the coding problem, and volatility of the ratings of thecontestants. In another embodiment, the method includes determining amaximum rating increase for the contestant. In yet another embodiment,the method includes estimating a probability of awarding a contestantmore points than other contestants in the competition. The determinationof a rating for the contestant may also include adjusting of the ratingin smaller increments as frequency that the contestant has been ratedincreases.

In yet another aspect of the invention, a method for providingcontestant information to a potential employer includes the steps ofreceiving contestant information from contestants for entry into on-linecoding competitions and providing an on-line coding competition betweencontestants. The method also includes assigning a rating to eachcontestant based on performance in the on-line coding competition andproviding some or all of the contestant information and the assignedratings of one or more contestants to a potential employer of thecontestant. In one embodiment, the method also includes awarding pointsto the contestant based on performance in the on-line competition. Thecontestant information may include such information as a home address,phone number, work address, e-mail address, professional experience,work history, and certain demographic information of the contestant. Inanother embodiment, a rating is assigned to a contestant based on thepoints awarded to the contestant.

In another aspect, a method for awarding a prize to a contestantincludes providing an on-line coding competition among contestants andawarding a point value to each contestant based on performance in thecompetition. The method also includes awarding prizes to a subset of thecontestants who have been awarded a point value above a predeterminedprize point value. The prize awarded depends on the points awarded withrespect to the predetermined point value. The prize can include, but maynot be limited to a monetary award, an interview with a prospectiveemployer, a tangible good, a job offer, or some combination.

In another aspect, a method for advertising in a coding competitionincludes receiving contestant information from contestants for entryinto on-line coding competitions. The method also includes the steps ofproviding one or more on-line coding competitions between contestantsand providing each contestant with one or more advertisement for viewingduring the competition. The method additionally includes tracking theviewing time of each advertisement by each contestant and providingcontestant information and the tracked viewing time of the advertisementto an advertiser.

In some embodiments, the contestant information is provided to theadvertiser in the aggregate. For instance, the server provides theinformation that twenty coders having a rating above a predeterminednumber viewed the advertisement for a total time of thirty minutes. Inother embodiments, the server provides advertisement to contestantsbased on contestant information. For instance, the server may provide anadvertisement to contestants having Java programming experience whileproviding another advertisement to contestants having C++ programmingexperience.

In one aspect, an apparatus for providing a coding competition includesa web server communicating with a web browser. The web browser is usedby a contestant to receive a competition request and enable thecontestant to enter the coding competition using client software. Theapparatus also includes a client interface server communicating with theclient software. The client interface server enables transmission of acoding problem to the client software and reception of computer code inresponse to the coding problem. The apparatus additionally includes anapplication server in communication with the web server and the clientinterface server. The application server interprets client requests,competition requests, and/or the computer code. The apparatus alsoincludes a contest server in communication with the application server.The contest server compiles the computer code submitted by contestants,executes the computer code, tests the computer code, and/or computes arating for the contestant.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. Also, the drawings are notnecessarily to scale, emphasis instead generally being placed uponillustrating the principles of the invention.

FIG. 1 is a block diagram of an embodiment of a coding competitionsystem according to the invention.

FIG. 2 is a block diagram of an embodiment of a server in the codingcompetition system of FIG. 1.

FIG. 3 is a more detailed block diagram of an embodiment of the serverof FIG. 2.

FIG. 4 is a flowchart of the operation of an embodiment of the serverfor a coding competition.

FIG. 5 is a more detailed flowchart of the operation of an embodiment ofthe server for a coding competition.

FIG. 6 is a block diagram of an embodiment of the evaluation performedby the server of a computer code's response to test data.

FIG. 7 is a flowchart of the operation of an embodiment of the serverfor a qualifying test before the coding competition of FIG. 4.

FIG. 8 is a flowchart of the operation of an embodiment of the serverfor completion of the qualifying test of FIG. 7.

FIG. 9 is a flowchart of the operation of an embodiment of the serverfor assigning a rating in the coding competition.

FIG. 10 is a flowchart of the operation of an embodiment of the serverfor attracting sponsorship of the coding competition.

FIG. 11 is a flowchart of the operation of an embodiment of the serverfor awarding a prize in the coding competition.

FIG. 12 is a chart of an embodiment of the progress of participants fromelimination rounds of competition to a national round of a codingcompetition.

FIG. 13 is an example screen shot of an embodiment of a web pagedisplayed during the coding competition.

FIG. 14 is an example screen shot of an embodiment of a coding problemselection screen displayed during the coding competition.

FIG. 15 is an example screen shot of an embodiment of a coding windowdisplayed during the coding competition.

FIG. 16 is an example screen shot of an embodiment of a challenge windowdisplayed during the coding competition.

DETAILED DESCRIPTION

Referring to FIG. 1, in one embodiment, a coding competition system 100includes at least one server 104, and at least one client 108, 108′,108″, generally 108. As shown, the coding competition system 100includes three clients 108, 108′, 108″, but this is only for exemplarypurposes, and it is intended that there can be any number of clients108. The client 108 is preferably implemented as software running on apersonal computer (e.g., a PC with an INTEL processor or an APPLEMACINTOSH) capable of running such operating systems as the MICROSOFTWINDOWS family of operating systems from Microsoft Corporation ofRedmond, Wash., the MACINTOSH operating system from Apple Computer ofCupertino, Calif., and various varieties of Unix, such as SUN SOLARISfrom SUN MICROSYSTEMS, and GNU/Linux from RED HAT, INC. of Durham, N.C.(and others). The client 108 could also be implemented on such hardwareas a smart or dumb terminal, network computer, wireless device,information appliance, workstation, minicomputer, mainframe computer, orother computing device, that is operated as a general purpose computeror a special purpose hardware device solely used for serving as a client108 in the competition system 100.

Generally, clients 108 are operated by contestants and are used bycontestants to participate in a coding competition. In variousembodiments, the client computer 108 includes either a web browser 116,client software 120, or both. The web browser 116 allows the client 108to request a web page (e.g. from the server 104) with a web pagerequest. An example of a web page is a data file that includes computerexecutable or interpretable information, graphics, sound, text, and/orvideo, that can be displayed, executed, played, processed, streamed,and/or stored and that can contain links, or pointers, to other webpages. In one embodiment, a user of the client 108 manually requests aweb page from the server 104. Alternatively, the client 108automatically makes requests with the web browser 116. Examples ofcommercially available web browser software 116 are INTERNET EXPLORER,offered by Microsoft Corporation of Redmond, Wash., and NETSCAPENAVIGATOR, offered by Netscape Communications Corporation of MountainView, Calif.

In some embodiments, the client 108 also includes client software 120.The client software 120 provides functionality to the client 108 thatallows a contestant to participate in a coding competition. The clientsoftware 120 may be implemented in various forms, for example, it may bein the form of a Java applet that is downloaded to the client 108 andruns in conjunction with the web browser 116, or the client software 120may be in the form of a standalone application, implemented in amulti-platform language such as Java or in native processor executablecode. In one embodiment, if executing on the client 108, the clientsoftware 120 opens a network connection to the server 104 over thecommunications network 112 and communicates via that connection to theserver 104. The client software 120 and the web browser 116 may be partof a single client-server interface 124; for example, the clientsoftware can be implemented as a “plug-in” to the web browser 116.

A communications network 112 connects the client 108 with the server104. The communication may take place via any media such as standardtelephone lines, LAN or WAN links (e.g., T1, T3, 56 kb, X.25), broadbandconnections (ISDN, Frame Relay, ATM), wireless links, and so on.Preferably, the network 112 can carry TCP/IP protocol communications,and HTTP/HTTPS requests made by the web browser 116 and the connectionbetween the client software 120 and the server 104 can be communicatedover such TCP/IP networks. The type of network is not a limitation,however, and any suitable network may be used. Typical examples ofnetworks that can serve as the communications network 112 include awireless or wired ethernet-based intranet, a local or wide-area network(LAN or WAN), and/or the global communications network known as theInternet, which may accommodate many different communications media andprotocols.

A computer coding contest provider operates a contest server 104, whichinteracts with clients 108. The server 104 is preferably implemented onone or more server class computers that have sufficient memory, datastorage, and processing power and that run a server class operatingsystem (e.g. SUN Solaris, GNU/Linux, MICROSOFT WINDOWS NT). Other typesof system hardware and software than that described here could also beused, depending on the capacity of the device and the number ofcontestants and the size of the contestant base and the competitions.For example, the server 104 may be part of a server farm or servernetwork, which is a logical group of one or more servers. As anotherexample, there could be multiple servers 104 that may be associated orconnected with each other, or multiple servers could operateindependently, but with shared data. As is typical in large-scalesystems, application software could be implemented in components, withdifferent components running on different server computers, on the sameserver, or some combination.

Referring to FIG. 2, in one embodiment, a server 104 includes a clientcommunication module 206 that is the interface for communication withclients 108. The client communication module 206 can be implemented assoftware running on one or more servers, or may be implemented as astand-alone server. In some embodiments, the client communication module206 can provide an interface both to client software 120 and to a clientweb browser 116, so that, for example, a web browser 116 can be used bya contestant to access performance information, or to observe theprogress of other contestants, and so on, while the client software 120can be used for participating in a coding competition. The interface toeach of the client software 120 and the client web browser 116 can beimplemented separately or in combination. In other embodiments, theclient communication module 206 can also communicate using otherprotocols or mechanisms.

The client communication module 206 communicates with the applicationserver 212, which provides the main programming logic for the operationof the system. In one embodiment, the application server 212 isimplemented as one or more application programs running on a serverclass computer, which may be the same or different computer as theclient communication module 206. The application server 212 receivesqueries from contestants via the client communication module 206. Theapplication server 212 also provides the coding competitions. Forexample, the application server 212 tracks the activity in the virtualrooms of a coding competition, and provides the coding problems for thecompetition. As another example, in some embodiments the applicationserver 212 may provide a qualifying test, which is a preliminaryrequirement for participating in a coding competition as a contestant.

The server 104 also includes a contest server 216. The contest server216 handles computationally intensive tasks associated with the codingcompetition, for example, the compilation and testing of code submittedby contestants in response to a coding problem. In one embodiment, thecontest server 216 also assigns ratings to contestants. Generally, theinterface between the application server 212 and the contest server 216is designed such that the computationally intensive tasks are queued forprocessing on the contest server 216, thereby allowing the applicationserver 212 to be responsive, even when loaded by one or more activecoding competitions.

The server 104 also includes a database server 220, which stores datarelated to the coding competitions in one or more databases. Forinstance, the database server 220 stores competition information,statistics about the contestants, contestant information, serveravailability, and web traffic information. The database server 220provides data to the application server 212. An example of the databaseserver 220 is an INFORMIX DYNAMIC SERVER, offered by INTERNATIONALBUSINESS MACHINES of Armonk, N.Y.

Referring to FIG. 3, in one embodiment, the client communication module206 (FIG. 2) includes an interface server 204, which provides aninterface for communications with the client software 120. In oneembodiment, the client interface server 204 is a software moduleexecuting on a server computer 104. In another embodiment, the clientinterface server is a stand-alone server computer in communication withother components of the server 104, such as the application server 212.In one embodiment, the client interface server 204 communicates with theclient software 120 using a messaging protocol that runs overTransmission Control Protocol/Internet Protocol (TCP/IP) forcommunications between the client software 120 and the client interfaceserver 204. The client interface server 204 is designed to maintaincommunications with the client software 120 of a large number of clients108.

The client interface server 204 includes an interface connection 304,which provides the actual interface to the client software 120. Theinterface connection 304 receives messages from the client software 120and provides them to an interface manager 308 and an interface serverprocess 312. Likewise, the interface manager receives messages from theinterface server manager 308 and the interface server process 312 andcommunicates those messages to the client software 120.

The interface server manager 308 manages the allocation of the interfaceconnection 304 and the interface server process 312 to each clientsoftware 120. Each time a client 108 (via client software 120) connectsto the server 104 (via client interface server 204), individualprocesses are assigned that will exist for the life of the client'sconnection. The interface server manager 308 handles this assignmentprocess. The interface server manager 308 also stores any informationregarding the number and nature of each of the client connections.

The interface server process 312 is the interface from the clientinterface server 204 to the application server 212. The interface serverprocess 312 communicates with the client software 120 at a higher levelthan the interface connection 304. The interface server process 312 alsocommunicates with the interface server manager 308. For example, theinterface server manager 308 may instruct the interface server process312 to obtain a particular application from the application server 212for use in the coding competition.

Thus, when a contestant wants to transmit a request to the server 104,the client software 120 transmits the request over the communicationsnetwork 112. The client software 120 communicates with the interfaceconnection 304 to enable communication with the interface server process312. The interface server process 312 receives the request from theinterface connection 304 and determines what needs to occur to handlethe request. The interface connection 304 also communicates the requestto the interface server manager 308 as necessary to properly manage andallocate resources of the client interface server 204 to handle therequest.

The interface server process 312 then transmits the request to theapplication server 212. The application server 212 includes businessobjects 316 to process the request. Each business object 316 manages oneor more clients 108. Business objects 316 communicate with the clientsoftware 120 via the interface server process 312. Business objects 316process requests from the client software 120. Examples of theimplementation of a business object 316 include, for instance, aComponent Object Model (COM) object and an Enterprise Java Bean (EJB)object.

If a request requires data access, the business objects 316 makerequests, via a data access layer 320, to the database server 220. Ifthe requests require processing by the contest server 216, the requestsare passed on to a message queue processor 324, which queues requestsfor operation by the contest server 216, and sends the results back tothe client software 102 via the interface server process 312.

In some embodiments, the client 108 (FIG. 1) can also communicate withthe server 104 via a web browser. In such an embodiment, thecommunication module 206 (FIG. 2) also includes a web server 208. Theweb server 208 delivers web pages to the client 108 and provides aninterface for communications between the web browser 116 and the server104. Preferably, the web server 208 is an enterprise class web server,such as APACHE from the APACHE FOUNDATION, INTERNET INFORMATION SERVERfrom MICROSOFT CORPORATION, or NETSCAPE ENTERPRISE SERVER from NETSCAPECOMMUNICATIONS CORPORATION, running on one or more server classcomputers, which may be the same or different computers than the clientinterface server 204. The web server 208 may be implemented as anycommunications device that is capable of serving web content to clientweb browsers 116 over the communications network 112, and communicatingwith the other parts of the server 104, such as a personal computer, webappliance and so on. The web server 208 and the client interface server204 are each in communication with the application server 212. The webserver 208 provides the interface for web browsers 116 to communicatewith the server 104. The web server 208 includes an HTTP listener 328that receives HTTP requests from the web browser 116 and transmits HTTPreplies to the browser 116. The HTTP listener 328 communicates with theapplication server 212 via a servlet engine 332.

The application server 212 communicates with web browsers 116 via theservlet engine 332. The servlet engine 332 uses Java servlets togenerate web pages in response to web page requests. An XSL processor336 interprets XSL style documents, which are used by the servlet engine332 to produce HTML documents. Data requests are made by the servletengine 332 to a business object 316, which makes data requests, ifnecessary, via the data access layer 320 to the database server 220. Theservlet engine 332 combines XML objects and XSL style files to produceHTML files, which are then communicated, via the web server 208, to theclient's browser 116.

The database server 220 includes one or more databases 340. The dataaccess layer 320 communicates with the database 340 to retrieve data.The database 340 can be implemented using commercially availabledatabase software, with standard interfaces. The various types ofinformation in the databases 340 can be stored in the same database orin different databases.

In one embodiment, the contest server 216 includes a code compilationengine 344. The code compilation engine 344 compiles contestants'computer code. Here, the term compilation is used in its broadest sense,and includes more than just translation of source code to object code.For example, if the competition system is used for hardware design, thecompiler can include tools to translate code into a circuit designformat suitable for simulation. The code compilation engine 344 caninclude any compiler. The choice of code compilation engine 344,however, affects the coding competition such that the compiler shouldsupports the competition coding languages and testing environment. Thus,in one embodiment, the server 104 may have the code compilation engine344 compile C or C++ code (e.g., a C++ compiler) using the GNU Ccompiler (GCC) from the FREE SOFTWARE FOUNDATION, Cambridge, Mass. Inone embodiment, the server 104 has the code compilation engine 344compile JAVA code using a compiler available from SUN MICROSYSTEMS ofMountain View, Calif. In another embodiment JAVA code is interpretedwithout compilation. In some embodiments, the code compilation engine344 includes or has access to compilers for the various coding languagesor formats to be used in the coding competition.

In one embodiment, the contest server 216 also includes a code testingengine 352, which tests the results of contestants' computer codeagainst test data, such as a predetermined test suite. The test inputmay come from various sources. Generally the contest server 216 runs,interprets, or simulates the test environment, such that the test datacan be provided as input, and the results obtained. Preferably, thecompilers used by the compilation engine 344 compile code such that thecode can be executed by the code testing engine 352. In one embodiment,the contest server 216 additionally includes a rating engine 348 tocompute and manage the contestants' ratings. The rating engine 348 mayimplement any algorithm, such as an algorithm based on statistics, thataccurately represents a contestant's skill level relative to othercontestants. Each of these tasks, code compilation, code testing, andrating generation, are computation intensive. Requests for theseoperations are queued by the message queue processor 324, and providedto the engines 344-352 in order of receipt. This allows the businessobjects 316 to complete the requests without blocking or waiting for theengines 344-352.

The de-coupling of these computation-intensive tasks in the contestserver 216 from the operation of the other servers allows forscalability of the competition system. Any combination of the servers,processors, or engines (e.g., web server 208, application server 212,message queue processor 324, rating engine 348) can be implemented onthe same or independent processors or computers relative to the otherservers, processors, or engines. Several instances of the contest server216 can execute in parallel (on the same or different computers) torespond to requests from the message queue processor 324. Additionalprocessing capacity can be added quickly to handle these tasks morequickly if the system comes under heavy load, as could happen during acompetition. Thus, a useful feature of this embodiment of the server 104is that it can reliably support a computation-intensive codingcompetition with large numbers of contestants.

Referring to FIG. 4, in one embodiment, the interaction of contestantswith the server 104 (via the client 108) can be summarized generally byfour steps. The contestant optionally registers with the server 104(STEP 404). Preferably, the contestant provides information aboutherself in the registration process. Examples of contestant informationcan include such information as the contestant's name, contactinformation (i.e., mail address, telephone number, email address), ausername or “alias” for use on the site, work experience, educationbackground, certain demographic information, specific technical skills,whether the contestant is currently seeking a job, and/or geographicpreferences of the contestant for employment. The alias is how thecontestant will be known on the system to other contestants andspectators. The registration may also include, for instance, eachcontestant completing a qualifying test, or qualifying round, asdiscussed in more detail below.

After registering, the contestant can participate in a competition. Inone embodiment, competitions are ongoing on the server 104 and heldcontinuously. In other embodiments, competitions are held at set times,or when announced by the competition administrator. To capture somecoders' attention and participation, for example, it may be better tohold competitions periodically after normal business hours. In oneembodiment, contestants enter a competition at any time. For example, acontestant may connect to the server 104 and may enter a competition atthat time if there are other suitable contestants available to compete.In one embodiment, this type of competition is referred to as a“pick-up” competition because a contestant can enter the competition atany time that other contestants are available. Competitions may bescheduled such that the server 104 automatically begins the competitionat the set time (or some predetermined delay after the predeterminedtime).

A competition first involves communication of the coding problem (STEP406). The server 104 transmits one or more computer coding problems toeach contestant via the client 108 in the competition. The contestantreviews the one or more coding problems and develops computer code as asolution to the coding problem (STEP 408). The contestant may select oneor more of the transmitted coding problems to solve. The contestant mayselect a first problem to solve, and then select another problem tosolve. For simplicity, one coding problem may be described, but it isintended that the principles can be extended to more than one.

In one embodiment, before a contestant submits computer code as a finalanswer to the coding problem, the contestant can submit the code forcompilation by the server 104, and for execution with test data providedby the contestant. In this way, the contestant can compile and/or runthe computer code to determine, for example, how well the computer codeoperates, the speed at which it operates, and the accuracy of thecomputer code. The contestants can additionally view, for example, theresult of compilation and the results of the execution with the testinput submitted by the contestant.

In one embodiment, the server 104 records the elapsed time taken by thecontestant in the coding stage. The server 104 starts a coding timerupon transmission of the computer problem to the contestants (via theclients 108) and ends the coding stage when contestant submits asolution, or when the coding timer reaches a predetermined coding stagetime (i.e. a time-out). Moreover, the server 104 may adjust thepredetermined coding time depending on the level of difficulty of thecomputer problem or problems.

In some embodiments, after submission of code, a challenge portion ofthe competition follows (STEP 412). During a challenge period, achallenging party can challenge a contestant's computer coding. Invarious embodiments, the challenging party can be one or more of:another contestant, a spectator of the coding competition, a judge orother party who has the assigned task of submitting test data to verifycode submissions, or the server 104. In one embodiment, the challengingparty challenges the computer code by submitting test data to the server104 for input to the executed computer code. The server 104 uses thetest data to determine the correctness of the computer code. With thetest data, the challenging party attempts to induce incorrect executionof the computer code, for example, generating an incorrect output, orhalting execution due to error. Based on the computer code's response tothe test data, the server 104 can determine if the computer codeexecution is acceptable. In some embodiments, the server 104 ends thechallenge when a challenge timer reaches a predetermined challenge time.Although described as independent timers, the timers such as thechallenge timer and the coding timer, may be implemented with a singletimer or multiple timers.

The server 104 may also conduct automated testing of the submitted code(STEP 416) in which the code is submitted to one or more automated testsfrom a predetermined test suite. In one embodiment, the server 104submits all submitted computer code to one or more automated tests. Theserver 104 may randomly choose a test from the predetermined test suiteor may choose a test based on, for example, a predetermined order or thecomputer code's performance with respect to an earlier challenge. Inanother embodiment, the challenge stage includes automated testing. Inanother embodiment, the automated testing precedes the challenge stage.

In one embodiment, a competition includes a contestant registering andthen participating in two or more competition rounds, where each roundconsists of problem communication (STEP 406), coding (STEP 408),challenge (STEP 412), and testing (STEP 416). Alternatively, acompetition round may include a subset of problem communication (STEP406), challenge (STEP 412), and testing (STEP 416). In some embodiments,the order of challenge (STEP 412) and automated testing (STEP 416) isreversed and in other embodiments, these steps overlap or take place inparallel. In one embodiment, a coding competition consists of apredetermined number of rounds, where the number varies from competitionto competition. Thus, a competition may only consist of one round. Acompetition with a number of rounds may also be referred to as atournament. As discussed below, the server 104 can additionally assignor adjust each contestant's rating after the entire competition iscomplete (e.g., after the round(s) of the competition are complete). Around can include communication of two or more problems, withoverlapping problem communication, challenge, and automated testing.

Referring to FIG. 5, in one embodiment, the server receives contestantinformation from the client (if it is the first competition for thecontestant) or the database (STEP 504). In some embodiments, the server104 does not enable the contestant to move to the next step untilrequired contestant information is received by the server 104. In someembodiments, the server 104 requests additional contestant informationfrom the client 108 at an appropriate time, such as during anintermission of a coding competition. In other embodiments, if acontestant has registered with the server 104 but has never competed ina competition, the contestant may not have to complete additionalinformation before competing in a competition. Thus, in one embodiment,all contestants provide a minimum of registration information prior tocompeting.

In another embodiment, the server 104 hosts a practice room. Thepractice room is a room in which one or more contestants can familiarizethemselves with the competition format (e.g., the format for answeringcoding problems). In one embodiment, inside the practice room, theserver 104 provides sample coding problems to one or more contestants.The contestant(s) attempt to develop code in response to these problems,which may help familiarize them with the competition system.

In one embodiment, the server 104 determines if enough contestants areavailable to begin the coding competition (STEP 508). If there are not,the server 104 waits until a predetermined number of contestants connect(via a client 108) to the server 104 for participation in a codingcompetition. In some embodiments, the server 104 may wait apredetermined time after establishing connections with two contestants(via clients 108) and then begin a competition thereafter. For instance,if the server 104 schedules a coding competition to begin on aparticular date and time, the server 104 may only wait a predeterminedamount of time (e.g., ten minutes) past the scheduled time beforebeginning the competition, as long as more than one contestant is goingto participate in the competition. If no contestants or only onecontestant submits contestant information at the scheduled time, theserver 104 may automatically cancel or postpone the coding competitionto a later date or time.

When a predetermined number of clients 108 associated with registered,qualified contestants are available to participate, the server 104transmits a coding problem to each contestant via the client 108 (STEP512). In one embodiment, the coding problem requires a contestant tocreate one or more computer programs, subroutines, and/or tasks. In oneembodiment, the coding problem involves developing code in one or morespecified languages or formats.

The server 104 can also impose a time limit on its acceptance ofresponses to the coding problem. In one embodiment, the server 104displays a response timer on the client 108 when the server 104transmits the coding problem. The server 104 sets the response timer toa predetermined response time to denote the time limit in which theserver 104 accepts computer code in response to the coding problem. Whenthe response timer expires, the contestant can no longer submit computercode.

In some embodiments, a contestant chooses one coding problem fromseveral transmitted coding problems. The server 104 transmits severalcoding problems to the client 108, where each coding problem has adifferent level of difficulty with respect to other coding problems. Ifa contestant submits a correct solution to a coding problem having ahigher level of difficulty, the more difficult choice can result in anawarding of more points relative to a correct solution to a codingproblem having a lower level of difficulty. After the contestant submitscode in response to one problem, the contestant may be able to work onanother coding problem. In another embodiment, the coding problemsoffered in a competition have the same degree of difficulty andtherefore have the same ascribed point value. In another embodiment,there is only one problem that must be responded to, with no choice.

The contestant may compile (if necessary) and test her code beforesubmitting the code to the server 104 as a final response. Thecontestant can thus validate her code before submission. By compilingand/or testing the computer code, the contestant may be able to, forexample, improve the computer code, determine errors in the computercode, determine the accuracy of the computer code, and determine theefficiency of the computer code. When the code is ready, the contestantthen submits her computer code to the server 104. The server 104receives the computer code from a contestant (STEP 516). The server 104compiles the code (STEP 520) if necessary. If the server 104 receivesexecutable or interpretable computer code (e.g. a JAVA program) from acontestant, there may be no need for a compilation step.

In one embodiment, the server 104 waits a predetermined amount of timeto receive computer code from each contestant. Once this predeterminedcoding time has elapsed, the server 104 does not accept codesubmissions. In other embodiments, the server 104 continues to acceptcode submissions, but the points available for the submission decreaseover time.

The server 104 verifies the correct execution of the submitted computercode (e.g., does it generate correct output, execute without errors inall cases, etc.). The server does this by obtaining test data (STEP524), which may come from various sources. In one embodiment, the server104 has predetermined test data to provide as input to the submittedcoding problems. Such test data is provided to the server with thecoding problem. In another embodiment, the server can generate test datausing information about the types of input that can be provided to thecomputer code (STEP 524). As described above, test data may also beprovided as a challenge by other contestants, or in some embodiments, byspectators or judges (FIG. 4, STEP 412). The server 104 may choose thecontestant's computer code to verify based on one or more factors, forexample, the order of submission of computer code to the server 104, theorder of test data provided to the server 104, the current ranking ofthe submitting contestant, the current amount of points awarded to thecontestant in this competition, as described below, randomly, or in apredetermined order. In one embodiment, in order to enable thisautomated verification of computer code responses, when test questionsare provided to the server 104 for use in a coding competition, areference computer code 636 as well as predetermined test data areprovided to the server 104 along with the test question. The referenceprogram 636 and the test data are stored with the test question for usein this manner.

The server 104 executes the code with the test data as input anddetermines the computer code's response (STEP 528). In one embodiment,the server 104 can display the results of the execution of the computercode with the test data as input to contestants and spectators. Inanother embodiment, the test data used and the results obtained arestored in a database for later viewing and verification.

In one embodiment, the server 104 generates test data and subjects thecode to the server's data as part of the automated testing stage. Evenif the server 104 previously provided or generated test data as achallenge, in one embodiment the server 104 inputs one or moreadditional test datum to the computer code.

The server 104 evaluates the response of the computer code (STEP 532).In one embodiment, the server 104 evaluates the response by comparingthe response to an acceptable response or a range of acceptableresponses. Further, if the contestant's computer code fails or produceserrors when executing with the test data as input, the server 104 maydeduct points from the submitting contestant. Moreover, if the computercode properly executes with the test data, the server 104 may awardpoints to the submitting contestant. In some embodiments, the server 104may also add or deduct points from a contestant who submits challengetest data for a particular computer code if the challenged computer codeexecutes correctly in response to the test data. In other words, thesystem will give points for a successful challenge and deduct points foran unsuccessful challenge of a contestant's computer code.

In one embodiment, in which the communications network 112 is theInternet, competitions take place in a “virtual forum” setting, in whichspectators can observe, via the computer network, the activity in avirtual competition room. Spectators may be users of a client 108 thathave registered with the server 104 who are not participating in aparticular coding competition, or may be observers who have not yetregistered with the server 104. In some such embodiments, besides theserver 104 and other contestants competing in the coding competition,spectators (either locally or via the network) can participate in thecompetition by submitting test data to the server 104 for input into thecontestants' submissions. Thus, spectators can help judge the worthinessof the submissions.

Also referring to FIG. 6, in one embodiment, the server 104 usesreference code 636 to verify the operation of a contestant's computercode 640. Preferably, the reference program 636 is a predeterminedsolution to a coding problem. Test data 639 is provided as input both tothe reference program 636 and the contestant's code 640. The test datamay be predetermined data stored by the server 104, data that isgenerated by the server 104, data provided by another contestant, ortest data provided by another party, such as a spectator, judge,administrator, etc. The reference program 636 produces a referenceresponse 638 during (or following) execution with the test data asinput. The contestants' computer code 640 also produces a response 644when it is executed with the test data as input. The server 104 thencompares the two responses 638, 644 to determine the correctness of thecomputer code response 644. The server 104 may also measure the timethat the reference program 636 and the contestant's computer code 640take to produce an associated response 638, 644 to determine theefficiency of the contestant's computer code 640. This comparison may berepeated for various input test data, with each test input coming fromthe same or a different source.

Referring again to FIG. 5, following the evaluation of the contestant'scomputer code, the server 104 awards points to the contestant (STEP536). In one embodiment, each coding problem has a maximum point valueassociated with it. In one embodiment, the server 104 awards less pointsthan the maximum point value that the coding problem is worth. Inparticular, the server 104 awards points to a contestant based on, forexample, some or all of the correctness of the contestant's coderesponse 644 (e.g. compared with the reference response 638), theaccuracy of the contestant's computer code response 644, the speed thatthe computer code 640 produces the computer code response 644, and/orthe recorded time at which the server 104 received the computer code 640from the contestant.

In one embodiment, the number of points awarded depends on the level ofdifficulty of the coding problem. The server 104 may also adjust thepoint value awarded to a contestant based on the time that the server104 receives the contestant's computer code 640 compared to the timethat the server 104 provides the coding problem. In one embodiment, thetotal number of points awarded to a contestant is as shown in Equation1:Total points awarded=MP*((0.3+(0.7*TT ²))/(10*PT ² +TT ²))  Equation 1

In Equation 1, PT is the time a contestant spends coding a problem(i.e., producing the computer code), TT is the total time allocated forcoding all computer problems in this part of the competition, and MP isthe maximum points available for the computer coding problem. In oneembodiment, the maximum number of points available is a frame ofreference for the server 104 that is not practically achievable by acontestant. More specifically, for a contestant to be awarded themaximum number of points, a contestant would have to spend almost zeroseconds reading the coding problem, coding the computer code 640, andsubmitting the computer code 640 to the server 104. For example, ifthere is one problem that is allocated 600 seconds (10 min.), and ittakes a contestant 300 seconds (5 min.) to code the problem, thenTT=600, PT=300, and approximately 52% of the total points MP will beawarded if the code executes correctly.

In one embodiment, the contestant receives this number of points (e.g.52% of the total available) for the computer code if the computer codeexecutes correctly with all test data, and the contestant receives nopoints if the code does not execute correctly with all test data. Inother embodiments, partial credit is given for partially correct code.For example, in one embodiment, the code receives the number of pointsbased on a percentage of test data. In another embodiment, the number ofpoints received is scaled based on the percentage of test data withwhich the code executes correctly. In some embodiments, such as when thecompetition is a tournament consisting of multiple rounds, the server104 tracks the points awarded to each contestant for each round. Theserver 104 may also calculate a running total for each contestant,summing the points awarded in each completed round of the competition.This running total can be used to eliminate low-scoring contestants.

In one embodiment, after the completion of one or more competitionrounds, the server 104 assigns a rating to the contestant (STEP 540).Typically the assignment of the rating occurs when a competition ortournament is complete. For example, a competition may consist of oneround, as described above with respect to FIG. 4, or may consist ofmultiple rounds, such as in a tournament. Thus, in the embodiment inwhich the competition consists of one iteration of the problemcommunication, coding, challenge, and testing steps, or more than oneround, the server 104 assigns a rating to each contestant after awardingtotal points to each contestant. In one embodiment in which thecompetition includes multiple rounds (e.g., a tournament), the server104 assigns a rating to each contestant after the contestants completethe tournament.

The ratings provide a metric that can be used to chart a programmer'sskill, knowledge, and ability. As described in more detail below, in oneembodiment the server 104 bases the contestant's rating on the totalnumber of points awarded to the contestant during the competition (e.g.,having one or multiple rounds). The rating can be a score, a grade, orany other type of rating technique.

Referring to FIG. 7, in one embodiment, a contestant undergoes aqualifying test as part of the registration process. The qualifying testcan take place at the time of registration, or at a later time. Duringthe qualifying test, the server 104 transmits one or more qualifyingtest questions to a contestant (STEP 704). In one embodiment, thecontestant has to complete the qualifying test in a predeterminedqualification time, which the server 104 displays on the client 108. Inone embodiment, the predetermined qualification time is a time for theentire qualification test. Alternatively, the predeterminedqualification time is a time requirement for one qualifying testquestion. In one embodiment, if the predetermined qualification timeelapses, the server 104 transmits the next question in the qualifyingtest. In another embodiment, the server 104 terminates the qualifyingtest. The server 104 may also time both the entire qualifying test andan answer to each qualifying test question.

The questions may be, for example, multiple choice questions, questionsrequiring one word answers, or a question that requires that thecontestant write computer code. Preferably, the questions arecomputer-related and correct response indicates a base level of skill.The questions may, however, be related to any subject, such as, forinstance, mathematics, science, literature, history, or logic. Moreover,in one embodiment, the qualifying test questions have varying degrees ofdifficulty. Further, the server 104 may or may not display these degreesof difficulty to the contestant, and the contestant may, before viewingthe qualifying test question, have a choice of which qualifying testquestion to answer based on its level of difficulty.

The contestant (via the client 108) answers each qualifying testquestion and then submits the answer to the server 104. The server 104receives the answer (STEP 708) from the client 108. In one embodiment,the server 104 evaluates the received answer (STEP 712). In anotherembodiment, once the contestant answers the test question, the client108 (e.g., the web browser 116 or the client software 120) automaticallytransmits the answer to the server 104 and the server 104 provides theclient 108 with the next qualifying test question (STEP 704).

During the evaluation of the received answer, the server 104 may, forinstance, determine the value of the received answer based on apredetermined set of correct answers and the speed at which thecontestant responded to the qualifying test question. The server 104 mayadditionally determine a number of points to award to the contestant forthe contestant's answer to the qualifying test question, or qualifyingtest question score, based on the previous determinations (e.g.,correctness of the received answer). After this evaluation, in oneembodiment, the server 104 notifies the contestant about the result(s)of the evaluation of the contestant's answer, such as, for example, thecorrectness of the contestant's answer or the qualifying test questionscore. In one embodiment, the server 104 notifies the contestant via aqualification results window displayed on the client 108 as thequalification test progresses. In another embodiment, the server 104only notifies the contestant about the result(s) of the evaluation ofthe qualifying test at its completion.

The server 104 determines if the qualifying test is complete (STEP 720).The server 104 establishes that the qualifying test is complete basedon, for example, when the predetermined qualification time for thequalifying test elapses, when the server 104 receives an answer to thefinal qualifying test question, or when a predetermined time for thefinal qualifying test question elapses. If the server 104 determinesthat the qualifying test is not complete after the reception of thecontestant's answer to the current qualifying test question, the server104 subsequently transmits the next qualifying test question in STEP704. If, however, the server 104 determines that the qualifying test iscomplete after the reception of the contestant's answer to the currentqualifying test question, the server 104 determines if this contestantqualifies for a coding competition.

Thus, after determining that the qualifying test is complete, the server104 may then evaluate the answers submitted by the contestant (STEP724). In this embodiment, the server 104 can then notify the contestantof the results of the evaluation (STEP 728). This notification may be anotification as to the correctness of each question, the time taken toanswer each question, the time taken to complete the qualifying test,the level of difficulty of each test question, the qualifying testquestion score, total points awarded for the qualifying test, orqualifying test score, and/or a minimum number of points needed toqualify for the coding competition. The points awarded per question maydepend on if the answer was correct, the time taken to receive thesubmitted answer, and the level of difficulty of the question. In oneembodiment, participants have the option of taking the qualifying test asecond time should they want to try to improve their score to increasetheir chances of qualifying.

In one embodiment, after determining that the qualifying test iscomplete, the server 104 allows the user to enter a coding competition.In this embodiment, the qualification test is an opportunity for a newcontestant to become familiar with the competition environment andpractice before a coding competition begins.

Referring to FIG. 8, in determining whether a contestant can participatein a coding competition, the server 104 looks at the qualifying testscore of the contestant, and the number of contestants alreadyparticipating.

The server 104 first determines if the qualifying test score is greaterthan a predetermined threshold qualifying test score (STEP 804). Thecontestant can not participate in the coding competition (STEP 808).

If the qualifying test score is greater than the predeterminedqualifying test score, the server 104 then increments a contestantcounter, which represents the number of contestants who have qualifiedfor the coding competition (STEP 812). If, however, the server 104determines that the contestant counter is less than or equal to thepredetermined maximum contestant count (STEP 816), the contestant isallowed to compete (STEP 824).

In one embodiment, the registration process includes the server 104seeding each contestant into a division. In one such embodiment, theserver 104 allocates two divisions, a first division for contestants whohave previously attained a rating greater than a predetermined divisionrating and a second division for contestants who have either notattained a rating (i.e., who have never competed before or have no dataassociated with a competition) or have a rating below the predetermineddivision rating. In one embodiment, the server 104 provides codingproblems having a lower degree of difficulty to the second divisionrelative to the problems provided to the first division.

For example, the server 104 could use a predetermined division ratingvalue of 1200. Thus, the server 104 seeds all contestants who have arating less than 1200 into the second division and all contestants whohave a rating greater than 1200 into the first division. Moreover, inone embodiment, the server 104 assigns all contestants again after thecompletion of a competition. Thus, after a competition, if a contestantpreviously in the first division has a rating below 1200 after thiscompetition, the server 104 places the contestant in the seconddivision. Further, the server 104 can select the prize that thecontestants are competing for depending on the division that thecontestants are competing in.

The server 104 may further divide up each division into one or more“rooms” or group of rooms. For example, the server 104 can divide up thesecond division into unrated rooms having only contestants who havenever been assigned a rating, and rated rooms having contestants whohave been assigned a rating below the predetermined division rating.

Referring to FIG. 9, the server 104 quantifies a contestant's codingabilities after the completion of some or all of a competition by ratingthe contestant. In one embodiment, the server 104 determines the ratingbased on the total points awarded to the contestant in the competition.

As above, the server 104 electronically communicates a coding problem toeach contestant (STEP 904). The server 104 then receives a contestant'scomputer code in response to the problem (STEP 908). The server 104 thenevaluates the received code (STEP 912).

Based on these evaluations of the code, the server 104 then awardspoints to the contestant (STEP 916). Once some or all of the competitionis complete (e.g., one round or multiple rounds), the server 104determines each contestant's rating (STEP 920). In one embodiment, todetermine a contestant's rating, the server 104 first determines acontestant's previous rating if the contestant has been ratedpreviously. Preferably, the server 104 retrieves the ratings of thecontestants from, for instance, the database server 220. The ratingincludes a rating value and a volatility value. The volatility valueindicates the consistency of the contestant's ranking over time.

In one embodiment, the server 104 calculates the average rating for thecontestants participating in a competition using Equation 2:

$\begin{matrix}{{AveRating} = \frac{\sum\limits_{i = 1}^{NumCoders}\;{Rating}_{i}}{NumCoders}} & {{Equation}\mspace{14mu} 2}\end{matrix}$The server 104 also calculates a competition factor using Equation 3:

$\begin{matrix}{{CF} = \sqrt{\frac{\sum\limits_{i = 1}^{NumCoders}\;{Volatility}_{i}^{2}}{NumCoders} + \frac{\sum\limits_{i = 1}^{NumCoders}\;\left( {{Rating}_{i} - {AveRating}} \right)^{2}}{{NumCoders} - 1}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$In Equations 2 and 3, NumCoders is the number of contestants in thecompetition and Rating_(i) is the rating without the volatility of theith contestant in the competition before the competition. Further,Volatility_(i) is the volatility of the ith contestant in thecompetition before the competition.

Next, the server 104 determines the skill of the contestant, as shownbelow in Equation 4.

$\begin{matrix}{{Skill} = \frac{{Rating} - 1200}{1200}} & {{Equation}\mspace{14mu} 4}\end{matrix}$The Rating is the contestant's rating before the competition, or aprevious rating. In one in embodiment, if the server 104 determines thatthe contestant has not been previously rated, the server 104 does notcalculate the skill. In a further embodiment, the server 104 assigns apredetermined skill level to a contestant (e.g. 1). The servercalculates the deviation of a contestant from other contestants usingEquation 5, where Volatility is the volatility of the contestant beforethe competition:

$\begin{matrix}{{Deviation} = \frac{Volatility}{1200}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

The server 104 then estimates the probability (WP_(i) for i from 1 toNumCoders) of the server 104 awarding a contestant a higher scorerelative to the other contestants in the competition. In one embodiment,the server 104 estimates this probability by using the contestant'sexpected performance rating on a range of good to bad days. The server104 models each contestant's performance as a normal distribution with amean (the rating) and standard deviation (the volatility). Using thismodel, the server 104 converts a contestant's performance into apercentage using the normal distribution, so that 50% representsexpected performance and 25% represents better three days out of four,etc. Then, to estimate the probability that the contestant beatscontestant i, the server 104 divides the range from 0% to 100% into 1%blocks, and the server 104 takes the midpoint of each block and computesthe corresponding performance. The win probability is the number ofblocks where the midpoint favors contestant i. If two contestants havethe same number of blocks in which the midpoint favors one contestant(i.e., a tie), the server 104 splits the block between both contestants.

In particular, the psuedo-code function to estimate the probability ofcontestant A beating contestant B, where f is the inverse of thestandard normal function, is shown below:

WinProbability( SkillA, DeviationA, SkillB, DeviationB) Set count, a, bto 0. Set fa = SkillA + f((a + .5)/100)*DeviationA. Set fb = SkillB +f((b + .5)/100)*DeviationB. While (a<100): If fa<fb: Add 100-b to count,Increment a, Set fa = SkillA + f((a + .5)/100)*DeviationA, If fa == fb: Add 100-b-.5 to count, Increment a, Increment b,  Set fa = SkillA +f((a + .5)/100)*DeviationA,  Set fb = SkillB + f((b +.5)/100)*DeviationB If fa>fb:  Increment b  Set fb = SkillB + f((b +.5)/100)*DeviationB Return count, where count is the win probability.

The server 104 then calculates the expected rank of the contestant usingEquation 6:

$\begin{matrix}{{ERank} = {\sum\limits_{i = 1}^{NumCoders}\;{WP}_{i}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$Where Wpi is the Win Probability of a coder against each other coder.The server 104 also calculates the expected performance of thecontestant using Equation 7:

$\begin{matrix}{{EPerf} = {- {\Phi\left( \frac{{ERank} - {.5}}{NumCoders} \right)}}} & {{Equation}\mspace{14mu} 7}\end{matrix}$In Equation 7, Φ is the inverse of the standard normal function. Theserver 104 then determines the actual performance of each contestant:

$\begin{matrix}{{APerf} = {- {\Phi\left( \frac{{ARank} - {.5}}{NumCoders} \right)}}} & {{Equation}\mspace{14mu} 8}\end{matrix}$

In Equation 8, ARank is the actual rank of the contestant in thecompetition based on score (1 for first place, NumCoders for lastplace). If the contestant's score was equal to another contestant, therank is the average of the positions covered by the tied contestants.The server 104 then calculates the “performed as” rating of thecontestant using Equation 9:PerfAs=OldRating+CF*(APerf−EPerf)  Equation 9The server 104 also determines the weight of the competition for thecontestant in Equation 10 and a cap for the contestant in Equation 11:

$\begin{matrix}{{Weight} = {\frac{{TimesPlayed} + 1}{{{.75}*{TimesPlayed}} + {.4}} - 1}} & {{Equation}\mspace{14mu} 10} \\{{Cap} = {150 + \frac{1500}{1 + {TimesPlayed}}}} & {{Equation}\mspace{14mu} 11}\end{matrix}$In Equation 10, TimesPlayed is the number of times the contestant hasbeen rated before. In one embodiment, to stabilize the higher ratedcontestants, the Weight of contestants whose rating is between 2000 and2500 is decreased 10% and the Weight of contestants whose rating is over2500 is decreased 20%. The server 104 then determines the new volatilityof the contestant using Equation 12 and the new rating of the contestantwith Equation 13:

$\begin{matrix}{{NewVolatility} = \sqrt{\frac{{Volatility}^{2} + {{Weight}*\left( {{CF}*\left( {{APerf} - {EPerf}} \right)} \right)^{2}}}{1 + {weight}}}} & {{Equation}\mspace{14mu} 12} \\{\mspace{79mu}{{NewRating} = \frac{{Rating} + {{Weight}*{PerfAs}}}{1 + {Weight}}}} & {{Equation}\mspace{14mu} 13}\end{matrix}$In one embodiment, if |NewRating−Rating|>Cap, the server 104 adjustsNewRating so that it is at most Cap different than Rating. Moreover, insome embodiments, the server 104 stores any of the above mentionedvalues, such as the Rating, Volatility, and Number of times previouslyrated. Additionally, in one embodiment, ratings of new contestants areprovisional before competing in a coding competition.

Thus, in the embodiment just described, the server 104 does not adjust acontestant's rating drastically from one performance in a codingcompetition if the contestant has performed and been rated in many othercoding competitions or rounds of coding competitions. The server 104 mayalso determine a maximum rating increase for each contestant to helpstabilize a contestant's rating from one coding competition to another.In one embodiment, the server 104 displays each contestant's ratings onthe web site of the coding competition. However, in another embodimentthe server 104 only enables registered contestants and sponsors to viewthe ratings of contestants.

TABLE 1 illustrates three examples of coding problems transmitted by theserver 104 to contestants. As shown in the three examples, codingproblems may include notes to assist the contestant in writing asolution to the problem. Also, coding problems may provide one or moreexamples of correct output of the computer code that solves the problem.

TABLE 1 Coding problem Example Value Implement a class Finances, whichcontains a function (100, 0, 100) −> 1  250 timeLeft. TimeLeft is passed3 int values. The initial value of (200, 1, 300) −> 0 the bank accountis initialValue. Interest is compounded (3453, 8, 200) −> 1200 monthlyat a rate of interest percent. Every month monthly dollars is taken outfrom the account (immediately after the Interest has been added).timeLeft must return an int that is the number of months the account cansupport the full monthly withdrawal. If the account will last longerthan 1200 months, the function should return 1200. The method signatureis: int timeLeft (int initialValue, int interest, int monthly). Note:initialValue and monthly are > or equal to 0 and < or equal to 100,000.Interest is > or equal to 0 and < or equal to 1000. Implement a classFraction, which contains a method If index = 3 and max  500getXSmallest. The method is passed two integers: an index denominator −4, all and a maximum denominator. The program should find all thefractions of the fractions of the form a/b where b is less than or equalto the form a/b, are ¼, ⅓, maximum denominator and a/b is less than 1and then return ½, 2/4, ⅔, ¾, and the index-th smallest of thesefractions. The method should the method should return the fraction as adouble value, rounded to 3 decimal return the third points (use thesupplied formatDouble function). Note: If the smallest, as a double:index value is > the number of fractions that are less than 1, the 0.5.method should return 1. Fractions should not be simplified. ½ and 2/4are two unique fractions, despite their equal double values. The classand method must be declared as public. The index must be between 1 and70, inclusive. The max denominator must be between 2 and 12, inclusive.Implement a class Numbers that contains a method num = 8: 8 can be 1000pairwisePrimes. The method inputs an int (num) and returns written as1 + 2 + 5 the number of ways the number can be written as the sum of and1 + 3 + 4 and the three distinct integers that are pairwise relativelyprime; that is, method should return no pair of the three integers has acommon factor > 1. Note: 2. num = 18: the num is > 0 and < or equal to40. One of the three distinct method should return integers can be 1;Numbers are Pairwise Relatively Prime if 14 they share no factor GREATERthan 1.

The coding problems shown in TABLE 1 have different levels ofdifficulty. These levels of difficulty are depicted as the differentpoint values of each coding problem. For instance, a contestant canchoose a 250 point coding problem to earn a maximum of 250 points for acorrect solution. However, another contestant may choose a 1000 pointcoding problem, which has a higher level of difficulty relative to a 250point question, to earn a maximum of 1000 points for a correct solution.Thus, in one embodiment, a contestant having a higher degree of skillcan amass points more rapidly than a contestant having a lower degree ofskill by selecting coding problems with higher point values (andtherefore higher levels of difficulty). In one embodiment, a server 104assigns a higher rating to a contestant that amasses a higher totalpoint value.

In one exemplary embodiment, multiple coding problems are available toall of the contestants in a virtual room. The contestants are given achoice of which problem to work on. Once a contestant views a problem,the timing on that problem begins. When the contestant completes thecode that is a response to one problem, and submits it to the server,the contestant can begin work on a second problem, and then the third,and so on, depending on the number of problems amount of time that isallocated to coding in the competition. In some embodiments, thecontestants do not see the problems, but only the point values, and thecontestants select a problem to work on based only on the point value ofthe problem. Thus, each contestant makes a judgement about his abilitiesin deciding which coding problem to work on first.

Referring to FIG. 10, the server 104 may use the ratings for manypurposes, including for attracting sponsors to the coding competition.The ratings provide a tangible, objective, and quantifiable way forcompanies to determine the skill level of a contestant. As above, theserver 104 performs the registration stage in which the server 104receives contestant information (STEP 1004). The server 104 then runs atleast one coding competition (STEP 1008) and assigns a rating to eachcontestant (STEP 1012).

The competition administrator can then use these competitions to attractsponsorship. In particular, the server 104 provides contestantinformation about one or more contestants to a first sponsor in exchangefor sponsorship (STEP 1016). Sponsorship can include many things, butgenerally includes providing financial support for the competitionand/or for the competition administrator. For example, in oneembodiment, a sponsor provides the prize money that is awarded tocontestants who have scored a higher number of point values relative toa predetermined number of other contestants. Although described below assponsoring the entire coding competition, a company can also sponsor apart of the coding competition (e.g., a round of a coding competition).In one embodiment, the server 104 causes the client to display a logoand other promotional information associated with a sponsor of thecoding competition on the contestant's web browser 116 (FIG. 1) orclient software 120 (FIG. 1). Moreover, the server 104 can provideselected contestant information and ratings to a sponsor.

In one embodiment, the competition administrator tailors the informationabout contestants that the server 104 provides to the sponsor. In thisembodiment, the server 104 transmits only portions of contestantinformation (STEP 1020). For example, the criteria may be a geographicpreference for employment, years of work experience, coding languagespecialty or other specific skills, and current employment status of thecontestant (e.g., active job hunting, passive job hunting).

The criteria may also be a rating range that a sponsor will consider foremployment opportunities. If the contestant is rated lower than theminimum rating for that sponsor, the server 104 does not provide thatcontestant's information to the sponsor. In other embodiments, theserver 104 receives the criteria from a sponsor at the time the companydecides to sponsor the coding competition. Thus, although the receptionof the criteria is shown in STEP 1020 after the server 104 provides thecontestant information to the sponsor, the server 104 may alternativelyreceive the criteria before the transmission of information to thesponsor in STEP 1016.

After the server 104 transmits contestant information for particularcontestants to a sponsor, the sponsor may then contact the contestantfor employment opportunities. In one embodiment, the sponsorcommunicates with the competition administrator to obtain contestantinformation for potential employment. Further, the sponsor communicateswith the competition administrator and the competition administratorthen communicates with the contestant regarding requested informationand employment opportunities. Thus, the competition administrator actsas an agent for the contestants to find employment.

In one embodiment, the server 104 provides information to the sponsorabout one or more contestants and the sponsor subsequently contacts thecontestant. In one embodiment, the sponsor contacts the contestantthrough a communication service (e.g. electronic messaging) provided bythe server 104. In this way, the actual identity of the contestant canbe hidden from the prospective employer, and the contestant can decidewhether to respond to the inquiries made by the sponsor. Additionally, acontestant who has a high rating relative to other contestants may usethat rating to bolster the contestant's negotiating power whennegotiating a starting salary and/or benefits with the prospectiveemployer.

In one embodiment, the competition administrator provides one or moreadvertisements to each contestant for viewing during the competition.The advertisements may be displayed on the web browser, the clientsoftware, or some combination. One or more advertisers supply theseadvertisements to the competition administrator. An advertiser may be asponsor of the coding competition and may further be a company or firmthat hires coders, develops software and/or hardware, and/or is involvedwith technology. The advertiser may also be the competitionadministrator. In one embodiment, the server 104 displays one or moreadvertisements on the client software 120. An advertisement may include,without limitation, a logo, a quote, a word, a paragraph, a picture, ora sound or video clip. The server 104 then tracks the viewing time ofthe advertisements by each contestant. For instance, the server 104 candetermines the amount of time that the client software displayed aparticular advertisement. The server 104 then provides contestantinformation about the contestants viewing the advertisement as well asthe tracked viewing time of the advertisement to the advertiser.

In one embodiment, the server 104 provides the same advertisement toeach contestant. The server 104 then provides contestant information tothe advertiser in the aggregate. The contestant information may include,for instance, the experience of the contestants viewing theadvertisement, the rating of each contestant, and/or the geographicpreference of each contestant. For example, the server 104 may providethat ten extremely skilled contestants having a rating above or equal to2000 viewed the advertisement for a total time of 30 minutes, twelvehighly skilled programmers having a rating above or equal to 1800 butbelow 2000 viewed the advertisement for 27 minutes, and twenty skilledprogrammers having a rating above or equal to 1500 but below 1800 viewedthe advertisement for 31 minutes. The server 104 may also provide a listof the time each contestant viewed the advertisement. In someembodiments, the server 104 provides other information about each of theabove-mentioned contestants, such as that they have a geographicpreference of Chicago, Ill.

In another embodiment, the server 104 provides different advertisementsto different contestants based on contestant information. For example,if an advertiser is a computer organization located only in New York anddevelops software using the Java programming language, the server 104displays this advertiser's advertisements to contestants havingcontestant information matching the above information (i.e., geographicpreference of New York, knowledgeable in the Java programming language).In one embodiment, the advertiser specifies to the competitionadministrator (or server 104) that the server 104 should only providetheir advertisement to coders having particular information as part oftheir contestant information. In another embodiment, the server 104increases the frequency of viewing opportunities of certainadvertisements based on information that the server 104 stores relatedto the particular advertiser.

Referring to FIG. 11, in addition to access to employment opportunities,a contestant who earned a large number of points in one or more roundsof a tournament relative to other contestants may be awarded one or moreprizes. As above, the server 104 provides the coding competition (STEP1104) and awards points to each contestant during (e.g., points earnedafter each round) and following (e.g., total points) the competition(STEP 1108). The server 104 determines if a contestant scoring aparticular number of points should receive a prize. The server 104 makesthis determination based on whether the awarded points are above apredetermined prize point value (STEP 1112) or whether the number ofpoints is greater than the number received by other contestants in theroom, division, etc.

If the contestant is to receive a prize, the server 104 notifies thecontestant. In one embodiment, different prizes are awarded to thecontestant based on the difference between the contestant's awardedpoints and the predetermined prize point value (STEP 1120). The prizemay be a tangible or intangible good. For example, a prize may include,without limitation, a monetary award, a vehicle, a computer, a vacation,and an interview with a potential employer. In a further embodiment, theprize is awarded to charity on behalf of the winner. The prize may bedelivered to the contestant in a variety of ways, such as mailing theprize to the contestant or having the contestant pick up the prize at apredefined location. In another embodiment, the coding competitionadministrator provides capital for one or more prizes.

In one exemplary embodiment, the prize distribution for a single roundof competition, is based on the division and the rooms. The contestantsare seeded into rooms based on their rating. The contestants in thehigher (i.e. lower-numbered) divisions and rooms is as shown in TableII. The prize money for first through third positions of each divisionand room is as follows:

TABLE II Room Contestants 1^(st) 2^(nd) 3^(rd) Division 1 1 9 635 537456 2 9 382 323 274 3 9 293 248 211 4 9 249 211 179 5 10 249 211 179 610 227 192 163 7 10 211 179 152 8 10 196 166 141 9 10 183 155 132 10 10172 145 123 11 10 165 140 119 12 10 158 133 113 Division 2 1 10 91 77 652 10 86 73 62 3 10 81 69 58 4 10 73 62 52 5 10 69 58 49 6 10 66 56 48 710 63 53 45 8 10 60 51 43 9 11 62 52 45 10 11 59 50 43 11 11 51 43 36Unrated 1 10 100 0 0 2 11 0 0 0 3 11 0 0 0

The server 104 may also provide “pick-up” competitions, wherecontestants communicate with the server 104 at any time and eachcontestant pays a competition fee to compete. The competitionadministrator can then distribute a portion of the total amountcollected (e.g., total amount collected less a percentage asadministration fee) to the winner of the pick-up competition.

In some embodiments, a series of coding competitions are used toidentify premier coders. Contestants participate in rounds, with thehigh scorers for each round participating in the next round. Thisenables initial participation by a large number of contestants, with thegoal of identifying the premier coders in that large number ofcontestants.

In one exemplary embodiment, a national tournament is held in the UnitedStates. In this embodiment, early rounds of the competition take placeover the Internet, with each of the contestants participating from home.Regional rounds take place in a central physical location for thatregion, and a national round is hosted in a large vacation-type settingsuch as Foxwoods in Ledyard, Conn.; Las Vegas, Nev.; or Orlando, Fla. Asponsor of the coding competition may receive in-person access tocontestants during the higher rounds, in the form of contactinformation, meals, meetings, and requests/performance of services (e.g.programming, publicity, etc.).

Referring to FIG. 12, in one embodiment, contestants can progress from afirst elimination round 1204 through the national championship 1220 of acompetition series. A predetermined number of contestants are invited tocompete in the first elimination round 1204, based on rating and/orother factors. In each round, the server 104 divides contestants intogroups each having a predetermined number of contestants. Eachcontestant in the group competes against the other contestants in thatgroup in a virtual room. In another embodiment, the server 104 populatesa room with a predetermined number of contestants who have similarratings. For example, a first virtual room may have the ten contestantswith the highest ratings, while another virtual room has ten contestantswho have never entered a coding competition before. This sort of roomseeding enables fairer competition between contestants. Further, in oneembodiment prize money is awarded to one or more contestants in eachvirtual room who, for instance, accumulate the highest point value forthat room. In some embodiments, an effort is made to allocatecontestants to virtual rooms by their geographic location, so that theycan participate in a regional competition with others from the sameregion.

In this embodiment, as the stakes increase (e.g., the amount of prizesawarded) from the first preliminary round 1208 to a national round 1220,so does the level of competition. Each round of the competition involvescoding, challenge and/or testing. Some of the contestants in the firstpreliminary round 1208 are eliminated during the first preliminary round1208, and so fewer number of contestants participate in the secondpreliminary round 1212. Likewise, still more contestants are eliminatedas the process continues into a regional round 1216, in which, in oneembodiment, the clients 108 and server 104 are located at one centralphysical location for that region. Finally, a predetermined number ofcontestants having the highest points in the regional round 1216 enter anational round 1220. The national round 1220 preferably takes place inone physical location (e.g., a vacation-type setting, or at a largecomputer conference). It is intended that this embodiment is exemplary,and that other embodiments might include different numbers of rounds,contestants, and locations.

Referring to FIG. 13, an example screen shot of an embodiment of avirtual “lobby” area 1300 is shown. In one embodiment, the lobby 1300 isthe main web page displayed on the client 108 during a codingcompetition. The lobby 1300 includes a status panel 1304 which contains,for instance, a contestant's login name, ranking, and status (i.e.,whether the contestant is coding or challenging). The lobby 1300 alsoincludes a main menu 1308 providing functions that can be used at anytime, such as a search function to search for a particular contestant.The lobby 1300 further includes an options area 1310, providing optionsto tailor the display of the lobby 1300. An active contest area 1312 isan area displaying the current active competitions (e.g., a practicecompetition, an actual competition). The lobby 1300 also includes a helparea 1316 for a user to retrieve help on a particular item, and acontest timer display 1320 which displays a relevant time related to thecoding competition. For instance, the contest timer display 1320 candisplay a start timer which counts down to the start of the nextcompetition. A legend key 1322 displays the contestants competing in thenext coding competition. The lobby 1300 also includes a chat area 1324which enables contestants in the virtual room to communicate with eachother. The lobby 1300 also enables a contestant to search for anothercontestant via a search box 1328. Additionally, a contestant informationwindow 1332 enables a contestant to find out certain information aboutanother contestant.

Referring to FIG. 14, in the embodiment of FIG. 13, a contestantcompeting in the coding competition selects one coding problem fromseveral coding problems that the server 104 transmits to each client108. An exemplary coding problem selection screen 1400 includes alisting 1404 of all virtual rooms that are active. The screen 1400 alsoincludes a tools window 1408. The coding problem selection screen 1400additionally includes a selection window 1412 where a contestant choosesa coding problem based on a displayed point value for that codingproblem. Once a contestant selects a coding problem, the server 104displays a coding window to enable a contestant to create computer codeto solve the coding problem.

Referring to FIG. 15, an exemplary embodiment of a coding window 1500includes a problem information window 1504 to display information aboutthe coding problem, such as the method name, class name, return type,and argument types. The coding window 1500 also includes a problemstatement window 1508 to display the coding problem. The coding window1500 additionally includes a coding area 1512 in which the contestantwrites computer code attempting to solve the coding problem. Moreover,the coding window 1500 includes an action bar 1516, where the action bar1516 enables the contestant to perform various actions on the computercode, such as testing the computer code, saving the computer code, andsubmitting the computer code to the server 104.

Referring to FIG. 16, in one exemplary embodiment, after completing thecoding stage, the server 104 executes the challenge stage and displays achallenge window 1600. The challenge window 1600 includes a viewingwindow 1604 that enables a contestant to view other contestants'computer code. The challenge window 1600 also includes a challengesubmission window to enable the submission of test data.

In one embodiment, the server 104 assigns teams for a team competition.This assignment may occur, for example, before a competition begins,after a predetermined number of rounds in a competition, or after onecompetition is complete and for future competitions. In furtherembodiments, the server 104 assigns teams randomly, based on previousratings, or based on previous points received by one or more contestant.In other embodiments, contestants form teams. In yet another embodiment,the server 104 elects at least two contestants to be team captains andthe team captains select contestants to be members for their team.Furthermore, the server 104 can allow a contestant to be traded from oneteam to another team. Examples of a trade that one team (e.g., the teamcaptain) can make for one contestant include, but are not limited to, amember of another team, points, and prizes.

A team competition can involve seasons, where teams compete for the bestrecord or point total over a predetermined period of time. Moreover, theserver 104 may include separate divisions for teams. In this embodiment,teams with an average rating above a predetermined team rating, forinstance, are placed into a higher division. The server 104 may alsoprovide a “handicap” for a team based on average rating advantage. Inone embodiment, the server 104 provides this handicap by awardingcontestants on the team with handicap points before the contestantbegins or during the contestant, without the contestant answering acoding problem.

In some embodiments, the server 104 enables corroboration betweencontestants on the same team, such that each contestant on a team canview the work and/or progress of other team members. In anotherembodiment, the server 104 aggregates the scores of the various teammembers (i.e., disallowing collaboration) to arrive at a team score. Inone embodiment, the server 104 provides a team competition in which someor all team members have to use the computer code created by other teammembers of the team to create the final computer code for submission.

In another embodiment, the contestants compete with respect to hardwarelogic design. The server 104 provides a coding problem which requestseach contestant to design of electronic circuitry and submit thecomputer code for the circuit design when complete. The coded circuitcan then be challenged using test data as input into a simulation of thecircuit. The execution of the code, then, is execution of a simulationof the circuit specified by the code. The server 104 can then awardpoints to the contestant based on, for instance, the circuit'sefficiency (e.g. number of components, projected implementation cost,maximum input and response speed, etc.), correctness in its responserelative to a predetermined solution, and success with respect to thetest data. Simple examples of a coding problem for a circuit would bedesign of an eight-bit a multiplication circuit (i.e., a circuit whichmultiplies two numbers) using specified logic gates in the digitalhardware domain, or the design of a low-pass filter in the analogdomain. Likewise, the system and methods of the invention can be appliedto coding in other computer and non-computer disciplines. If the resultcan be coded, and then the code executed or simulated then theprinciples described above apply. For example, in the field of civilengineering, with the proper tools, a portion of a bridge design (e.g.support truss) can be coded and the coded design can be tested with teststresses during simulation. In the field of aeronautical engineering, anairplane wing elevator design can be coded and the result subjected tovarious inputs (e.g. shifts in wind, angle, etc.) during simulation.Thus, with the appropriate coding tools, compilers, and executionenvironment, the method and systems described can be extended to codingin various disciplines.

Variations, modifications, and other implementations of what isdescribed herein will occur to those of ordinary skill in the artwithout departing from the spirit and the scope of the invention asclaimed. Accordingly, the invention is to be defined not by thepreceding illustrative description but instead by the spirit and scopeof the following claims.

What is claimed is:
 1. A system for automatically identifying solutionsto a problem statement by conducting a competition, the systemcomprising: a communication module for electronically inviting via acommunications network a plurality of potential participants toparticipate in a competition provided over the communications network,the competitions having a predetermined start time and end time, and forelectronically receiving indicia of interest from potentialparticipants; a contest server comprising at least one memory storingcomputer-executable instructions, and at least one processing unit forexecuting the instructions stored in the memory, wherein execution ofthe instructions results in an application comprising: (i) a submissionreceiving subsystem for electronically receiving software-basedsubmissions from participants after the start time and prior to the endtime of the competition, (ii) a testing engine for: (a) after the starttime and prior to the end time, automatically performing interimevaluations of the received software-based submissions by executing,using a processor, each received software-based submission using firsttest data as input and assigning points to the participants based on theinterim evaluations, (b) for each received software-based submissionfrom a participant, selecting respective second test data for theparticipant based on a performance of the received software-basedsubmission in at least one of the interim evaluations of the receivedsoftware-based submission, and (c) after the end time, automaticallyperforming final evaluations of the last software-based submissionreceived from each participant by executing, using a processor, eachlast software-based submission using the second test data selected forthe participant as input and assigning points to the participants basedon the final evaluations, and (iii) a winner identification subsystemfor automatically identifying as the winner the participant whosubmitted the submission that received the greatest number of pointsduring the final evaluations; and a data store for storing the receivedsubmissions.
 2. The system of claim 1, wherein the testing engineperforms evaluations by providing test data as input to a referenceprogram and the received submissions, and comparing output responsesfrom the reference program to output responses from the receivedsubmissions.
 3. The system of claim 1, wherein performing the finalevaluations comprises automatically determining a response of each lastsubmission to the second test data.
 4. The system of claim 3, whereinperforming the final evaluations further comprises automaticallydetermining the correctness of the determined response using the secondtest data.
 5. The system of claim 3, wherein performing the finalevaluations further comprises automatically determining the accuracy ofthe determined response using the second test data.
 6. The system ofclaim 3, wherein performing the final evaluations further comprisesautomatically determining the speed of the determined response bymeasuring the time taken to determine the response of the lastsubmission to the second test data.
 7. The system of claim 3, whereinperforming the final evaluations further comprises automaticallydetermining the accuracy of the determined response and the speed of thedetermined response using the second test data.
 8. The system of claim3, wherein performing the final evaluations further comprisesautomatically determining the the correctness of the determinedresponse, the accuracy of the determined response, and the speed of thedetermined response using the second test data.
 9. The system of claim1, wherein the second test data comprises the first test data andadditional test data.
 10. The system of claim 1, wherein at least one ofthe first test data and the second test data comprises challenge testdata submitted by at least one of another participant, a judge, and aspectator.
 11. A method, implemented on at least one computer, forautomatically identifying solutions to a problem statement by conductinga competition, the at least one computer comprising: at least one memorystoring computer-executable instructions; and at least one processingunit for executing the instructions stored in the memory, wherein theexecution of the instructions results in the at least one computerperforming the steps of: electronically inviting via a communicationsnetwork a plurality of potential participants to participate in acompetition provided over the communications network, the competitionshaving a predetermined start time and end time, and for electronicallyreceiving indicia of interest from potential participants,electronically receiving software-based submissions from participantsafter the start time and prior to the end time of the competition, afterthe start time and prior to the end time, automatically performinginterim evaluations of the received software-based submissions byexecuting each received software-based submission using first test dataas input and assigning points to the participants based on the interimevaluations, for each received software-based submission from aparticipant, selecting respective second test data for the participantbased on a performance of the received software-based submission in atleast one of the interim evaluations of the received software-basedsubmission, after the end time, automatically performing finalevaluations of the last software-based submission received from eachparticipant by executing each last software-based submission using thesecond test data selected for the participant as input and assigningpoints to the participants based on the final evaluations, automaticallyidentifying as the winner the participant who submitted the submissionthat received the greatest number of points during the finalevaluations, and storing the received submissions.
 12. The method ofclaim 11, wherein evaluations are performed by providing test data asinput to a reference program and the received submissions, and comparingoutput responses from the reference program to output responses from thereceived submissions.
 13. The method of claim 11, wherein performing thefinal evaluations comprises automatically determining a response of eachlast submission to the second test data.
 14. The method of claim 13,wherein performing the final evaluations further comprises automaticallydetermining the correctness of the determined response using the secondtest data.
 15. The method of claim 13, wherein performing the finalevaluations further comprises automatically determining the accuracy ofthe determined response using the second test data.
 16. The method ofclaim 13, wherein performing the final evaluations further comprisesautomatically determining the speed of the determined response bymeasuring the time taken to determine the response of the lastsubmission to the second test data.
 17. The method of claim 13, whereinperforming the final evaluations further comprises automaticallydetermining the accuracy of the determined response and the speed of thedetermined response using the second test data.
 18. The method of claim13, wherein performing the final evaluations further comprisesautomatically determining the the correctness of the determinedresponse, the accuracy of the determined response, and the speed of thedetermined response using the second test data.
 19. The method of claim11, wherein the second test data comprises the first test data andadditional test data.
 20. The method of claim 11, wherein at least oneof the first test data and the second test data comprises challenge testdata submitted by at least one of another participant, a judge, and aspectator.